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INTRODUCTION 

The objective of the proposed work is to develop an evaluation 
framework for NASA’s educational outreach efforts. We focus on 
public (rather than technical or scientific) dissemination efforts, 
specifically on Internet-based outreach sites for children.The out- 
come of this work is to propose both methods and criteria for 
evaluation, which would enable NASA to do a more analytic evalua- 
tion of its outreach efforts.The proposed framework is based on 
IRL’s ethnographic and video-based observational methods, which 
allow us to analyze how these sites are actually used. 

NASA’s Existing Evaluation Framework 

NASA already has in place an evaluation framework for educational 
programs: EDCATS (Education Computer Aided Tracking System). 
This system primarily provides external evaluation of face-to-face 
programs, such as classes, site visits, etc. It also has provisions for 
tracking hits and links for web sites. By external evaluation, we mean 
that the on-line questionnaires of the system capture basic demo- 
graphic information and participant feedback. For example, the 
student evaluation form includes: 

□ Name 

0 Address and contact information 

□ Characteristics, including 

□ Gender 

□ Nationality 

□ Disabilities 

□ Race 

□ Career goals 

□ Grade level 

□ School address 

□ Type of school 

□ Participation in past NASA activities 

□ Program content 

□ Type of teaching techniques/tools used in the program 

□ Program value 


□ Evaluation of presentation 

□ Program resources and tools used or distributed 

□ Other program benefits 

□ Process for announcing program, applying to program, 
running program 

□ Overall rating for program and program staff 

□ Additional comments 

Similar forms are available for teacher feedback. 

There are feedback forms for participants and for observers of 
NASA’s staged opportunities to chat with NASA experts.These 
forms ask participants to identify; 

□ Name and email address 

□ Teacher, student or other 

□ Number of people participating in chat from your location 

□ Number of computers used 

□ Featured NASA expert 

□ NASA project hosting the chat 

□ Date and time of chat 

□ Rate chat on scale of 5 

□ How did you decide which questions you would ask? 

□ Did the response answer your question. If not, explain 

□ Was the chat a valuable educational experience for you? 
Explain. 

□ General comments. 

In addition, there are forms available for evaluation of NASA web 
sites, which track: 

□ Number of WWW hits 

□ Data transfer volume 

□ Unique internet addresses accessing site in previous 

month > 

□ Number of redirections to other NASA web sites 

□ Number of search queries 

Our Evaluation Goals 

These are all necessary and important statistics to track. In addition, 
though it would be valuable to go beyond this to show more: how 
recipients use a site or a program, what physical and human re- 
sources a site requires to make it effective, how it could be made 
more appealing and useful to its users. At the same time, we share 
the caution of the EDCATS program in avoiding any attempt to 
measure direct effects on math or science test scores of NASA’s 
outreach programs. In any student’s education, there are so many 
other variables that it would be impossible to demonstrate a direct 
relation. 
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Further, NASA’s mission to communicate knowledge is not to directly 
improve math and science performance, but to develop an informed 
citizenry, who are knowledgeable and enthusiastic about NASA’s explor- 
atory and scientific missions.The NASA education vision, as set forth in 
the NASA Strategic Plan of 1 996 is: 

To promote the pursuit of educational excellence by 
involving the educational community in our endeavors to 
inspire America’s students, create learning opportunities, 
and enlighten inquisitive minds.' 

Therefore, we are attempting to evaluate the design of outreach sites 
and programs rather than their direct effects on test scores or other 
measures of student progress. Our efforts are directed at formative 
evaluation — evaluation designed to improve design, rather than at 
summative evaluation — evaluation measuring how recipients performed. 

Use ofVideo for Analysis of Design 

The major method for this project has been the analysis of videotapes 
of actual use of the outreach sites studied.Video analysis is the in-depth 
micro analysis of how people interact with one another, their physical 
environment and the documents, artifacts and technologies in that 
environment. (For information on the history, technology and application of 
this method, see Goodwin and Heritage 1 990, Jordan and Henderson 1995, 
McDermott et al 1 978, Roschelle 1994, Roschelle et al 1990.) Use of video 
provides a complete record that can be viewed over and over again and 
which permits the asking of new questions that may be completely 
different from those with which an investigation began.Video is also a 
powerful tool for designers, enabling them to see people actually trying 
to learn and use their designs. It provides information that is available in 
no other way. Thus, in this project, we are able to discover what web 
sites, activities and parts of an activity kids engage with. We are also able 
to see what kids do with the activities, how they shape of change them 
for their own interests.This provides important information\>r design- 
ers, since it allows them to see how their designs are used, ignored, or 
misused-all valuable information for the design cycle. 

Comparison of Evaluation Methods 

Below is a table summarizing the strengths and weaknesses of the 
methods discussed above. Note that EDCATS includes elements of 
several methods: collection of demographic statistics, survey question- 
naire, and interview methods, although its open-ended questions do not 
include the interactivity of face-to-face interviews, in which an inter- 
viewer may ask clarifying questions. 


Outreach Report 1999 


1 NASA Strategic Plan, January 1, 1 996, Page h 
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